Learning Accurate Integer Transformer Machine-Translation Models

We describe a method for training accurate Transformer machine-translation models to run inference using 8-bit integer (INT8) hardware matrix multipliers, as opposed the more costly single-precision floating-point (FP32) hardware. Unlike previous work, which converted only 85 multiplications INT8, leaving 48 out of 133 them in FP32 because unacceptable accuracy loss, we convert all INT8 without compromising accuracy. Tested on newstest2014 English-to-German translation task, our Base and Big yield BLEU scores that are 99.3–100% relative those corresponding models. Our approach converts matrix-multiplication tensors from an existing model into by automatically making range-precision trade-offs during training. To demonstrate robustness this approach, also include results INT6